Image processing device and image processing method for calculating distance information to a subject

ABSTRACT

Image processing device is image processing device that uses a plurality of images respectively having focusing positions different from each other to calculate distance information to a subject, and includes frequency converter, amplitude extractor, and distance information calculator. Frequency converter converts the plurality of images into frequency. Amplitude extractor extracts an amplitude component out of a phase component and the amplitude component of a coefficient obtained by converting the plurality of images into frequency. Distance information calculator calculates the distance information, by using lens blur data and only the amplitude component extracted by amplitude extractor out of the phase component and the amplitude component of the coefficient.

TECHNICAL FIELD

The present disclosure relates to an image processing device and animage processing method that use a plurality of images respectivelyhaving focusing positions different from each other to calculatedistance information to a subject.

BACKGROUND ART

In an image capturing device or the like, various techniques have beendevised for calculating without contact a depth of a certainthree-dimensional scene, that is, distance information to a subject fromthe image capturing device or the like. When those are roughlycategorized, there are an active technique and a passive technique. Inthe active technique, infrared light, ultrasound, or laser is emitted tothe subject, and based on a time until a reflected wave returns or anangle of the reflected wave, the distance information to the subject iscalculated. In the passive technique, the distance information to thesubject is calculated based on an image of the subject. Especially in acamera, the passive technique is widely used that does not require adevice for emitting the infrared light or the like.

Many techniques have been devised also in the passive technique. As oneof them, there is a technique called Depth from Defocus (hereinafterreferred to as DFD) in which the distance information to the subject iscalculated based on information of a blur, the blur changing in size andshape depending on a distance to the subject. The DFD has features suchthat a plurality of cameras is not required, and it is possible tocalculate the distance information to the subject by using a smallnumber of images.

Hereinafter, the principle of the DFD will be briefly described.

The DFD is a technique for calculating the distance information to thesubject, based on the information of the blur, from a plurality ofimages respectively having focusing positions different from each other.A captured image including the information of the blur (hereinafter,referred to as a blur image) is an image in which a point spreadfunction being a function of the distance to the subject is convolutedwith an all-in-focus image representing a state without a blur due to alens. Since the point spread function (hereinafter, referred to as thePSF) is the function of the distance to the subject, the distanceinformation to the subject can be calculated by detecting theinformation of the blur from the blur image, with the DFD). However, atthis time, the all-in-focus image and the distance information to thesubject are unknown. Since one formula relating to the blur image, theall-in-focus image, and the distance information to the subject isestablished for one blur image, blur images having different focusingpositions from each other are newly captured and new formulas areobtained. The obtained multiple formulas are solved, and the distanceinformation to the subject is calculated. Regarding a method forobtaining the formula and a method for solving the formula, variousmethods have been devised for the DFD, including the one in PTL 1.

CITATION LIST Patent Literature

PTL 1: JP H11-337313 A

SUMMARY

However, when the plurality of images respectively having focusingpositions different from each other is tried to be obtained, theplurality of images is obtained by changing focus of a lens system, sothat a time difference occurs between times to obtain the respectiveimages. When a position, a shape, or the like of the subject greatlychange within the time difference, positional deviation occurs in thesubject between the plurality of images, so that it may be difficult toaccurately calculate the distance information to the subject. That is,it may be difficult to use conventional DFD in capturing a still imageor a moving image of a fast-moving subject.

The present disclosure provides an image processing device and an imageprocessing method capable of accurately calculating the distanceinformation to the subject in capturing the still image or the movingimage of the fast-moving subject.

The image processing device in the present disclosure is an imageprocessing device that uses a plurality of images respectively havingfocusing positions different from each other to calculate distanceinformation to a subject, and includes a frequency converter, anamplitude extractor, and a distance information calculator. Thefrequency converter converts the plurality of images into frequency. Theamplitude extractor extracts an amplitude component out of a phasecomponent and the amplitude component of a coefficient obtained byconverting the plurality of images into frequency. The distanceinformation calculator calculates the distance information, by usinglens blur data and only the amplitude component extracted by theamplitude extractor out of the phase component and the amplitudecomponent of the coefficient.

In addition, the image processing method in the present disclosure is animage processing method that uses a plurality of images respectivelyhaving focusing positions different from each other to calculatedistance information to a subject, and includes converting the pluralityof images into frequency, extracting an amplitude component out of aphase component and the amplitude component of a coefficient obtained byconverting the plurality of images into frequency, and calculating thedistance information by using lens blur data and only the amplitudecomponent extracted out of the phase component and the amplitudecomponent of the coefficient.

The image processing device and the image processing method in thepresent disclosure are capable of accurately calculating the distanceinformation to the subject in capturing the still image or the movingimage of the fast-moving subject.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating a functional configuration of animage processing device in a first exemplary embodiment.

FIG. 2 is a block diagram illustrating a functional configuration of adistance information calculator in the first exemplary embodiment.

FIG. 3 is a diagram illustrating difference in accuracy for positionaldeviation between distance information calculated by a conventionaltechnique and distance information calculated by a technique of thepresent disclosure.

FIG. 4 is a block diagram illustrating a functional configuration of animage processing device in a second exemplary embodiment.

FIG. 5 is a block diagram illustrating a functional configuration of adistance information calculator in a third exemplary embodiment.

DESCRIPTION OF EMBODIMENTS

Hereinafter, exemplary embodiments will be described in detail withreference to the drawings as appropriate. However, more detaileddescription than necessary may be omitted. For example, a detaileddescription of already well-known matters and redundant description onsubstantially the same configuration nay be omitted.

This is for avoiding unnecessary redundancy of the followingdescription, and facilitating understanding by those skilled in the art.

Incidentally, the accompanying drawings and the following descriptionsare provided to enable those skilled in the art to fully understand thepresent disclosure, and are not intended to limit the claimed subjectmatter.

First Exemplary Embodiment

Hereinafter, a first exemplary embodiment will be described withreference to FIG. 1.

FIG. 1 is a block diagram illustrating a functional configuration of animage processing device in the first exemplary embodiment.

Image processing device 100 in the present exemplary embodiment is adevice that uses a plurality of images respectively having focusingpositions different from each other to calculate distance information toa subject. As illustrated in FIG. 1, image processing device 100includes frequency converter 110, amplitude extractor 120, and distanceinformation calculator 130.

Frequency converter 110 is a processor for converting the images intofrequency. That is, frequency converter 110, for the plurality of imagesrespectively having focusing positions different from each other,converts the images from an image space to a frequency space. The imagesare converted from the image space to the frequency space, wherebyaccuracy of distance information calculation can be improved. In thecase of the present exemplary embodiment, frequency converter 110converts the plurality of images from the image space to the frequencyspace. Here, description will be made assuming that, as the plurality ofimages, for example, a first image and a second image respectivelyhaving focusing positions different from each other are input tofrequency converter 110. Incidentally, the plurality of images may beinput in parallel or input serially. In addition, a method forconverting the images from the image space to the frequency space byfrequency converter 110 is not particularly limited, and for example,Fast Fourie Transform (FFT), Discrete Fourie Transform (DFT), and thelike are used.

Amplitude extractor 120 is a processor for extracting an amplitudecomponent out of a phase component and the amplitude component of acoefficient obtained by conversion into frequency by frequency converter110. In the case of the present exemplary embodiment, amplitudeextractor 120 extracts only amplitude components of coefficients of thefirst image and the second image obtained by conversion into thefrequency space. Specifically, since the coefficient obtained byconversion into the frequency space is a complex number, amplitudeextractor 120 extracts only the amplitude component by calculating anabsolute value of the coefficient. Incidentally, the coefficientobtained by conversion into the frequency space is also referred to as aconversion coefficient. In the case of the present exemplary embodiment,amplitude extractor 120 extracts a first amplitude component and asecond amplitude component that are amplitude components of the firstimage and the second image, respectively.

Distance information calculator 130 is a processor that uses lens blurdata and only the amplitude component extracted by amplitude extractor120 out of the phase component and the amplitude component of thecoefficient, to calculate the distance information to the subject.

Here, the lens blur data is an optical transfer function determined by aconfiguration of an optical system such as a lens and an aperture of acamera acquiring the images. Hereinafter, the optical transfer functionis referred to as the OTF.

FIG. 2 is a block diagram illustrating a functional configuration of adistance information calculator in the present exemplary embodiment.

As illustrated in FIG. 2, distance information calculator 130 includescost calculator 131 and distance determiner 132.

Cost calculator 131 is a processor that uses the lens blur data and thefirst amplitude component and the second amplitude component extractedby amplitude extractor 120, to calculate a cost of a distance assumed,for pixels of the images.

In the present exemplary embodiment, the DFD is used as a technique forcalculating the distance information.

In addition, the OTF is used as the lens blur data. The DFD, to estimatewhich distance corresponds to each pixel in the images, calculates thecost of the distance (Cost(d)) for a plurality of distances d assumed,for example. A formula for calculating the cost of the distance is shownby Formula 1 below.

$\begin{matrix}{\left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 1} \right\rbrack\mspace{365mu}} & \; \\{{{{Cost}(d)} = {{{abs}\left\{ {{ifft}\left( {\frac{{- \frac{{{{OTF}_{2}(d)}}^{2}}{{{{OTF}_{1}(d)}}^{2}}} + {\frac{{OTF}_{2}(d)}{{OTF}_{1}(d)} \cdot \frac{F_{2}}{F_{1}}}}{1 + \frac{{{{OTF}_{2}(d)}}^{2}}{{{{OTF}_{1}(d)}}^{2}}}{F_{1}}} \right)} \right\}^{2}} + {{abs}\left\{ {{ifft}\left( {\frac{{\frac{{OTF}_{2}(d)}{{OTF}_{1}(d)} \cdot \frac{F_{1}}{F_{2}}} - 1}{1 + \frac{{{{OTF}_{2}(d)}}^{2}}{{{{OTF}_{1}(d)}}^{2}}}{F_{2}}} \right)} \right\}^{2}}}}{{Amplitude}\mspace{14mu}{component}\mspace{14mu}{of}\mspace{14mu}{first}\mspace{14mu}{image}\text{:}\mspace{14mu}{F_{1}}}{{Amplitude}\mspace{14mu}{component}\mspace{14mu}{of}\mspace{14mu}{second}\mspace{14mu}{image}\text{:}\mspace{14mu}{F_{2}}}} & \left( {{Formula}\mspace{14mu} 1} \right)\end{matrix}$

Incidentally, Formula 1 is calculated based on Formula 2 below is adepth calculation formula considering image deviation. Here, the presentdisclosure can delete the phase component from the formula by applyingthe image deviation to one of the first image and the second image. Thatis, a solution can be obtained with only the amplitude component. It isnew knowledge to apply the image deviation itself and to introduce theimage deviation into the depth calculation formula. Conventionally, theimage deviation is not applied, so that the image deviation is notintroduced into the depth calculation formula, and as a result, thephase component remains in the formula even when Fourier transform isperformed, and the depth cannot be accurately calculated when there isthe image deviation.

$\begin{matrix}{\left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 2} \right\rbrack\mspace{365mu}} & \; \\{{{{Depth}\mspace{14mu}{calculation}\mspace{14mu}{formula}\text{:}\mspace{14mu}\hat{d}} = {\underset{d}{argmin}\left\lbrack {{{{{F_{0}\left( \hat{d} \right)}{{OTF}_{1}(d)}} - F_{1}}}^{2} + {{{{F_{0}\left( \hat{d} \right)}{{OTF}_{2}(d)}{OTF}_{shift}} - F_{2}}}^{2}} \right\rbrack}}{{{First}\mspace{14mu}{image}\text{:}\mspace{14mu} F_{1}} = {F_{0} \cdot {OTF}_{1}}}{{{Second}\mspace{14mu}{image}\text{:}\mspace{14mu} F_{2}} = {F_{0} \cdot {OTF}_{2} \cdot {OTF}_{shift}}}{{{Image}\mspace{14mu}{deviation}\text{:}\mspace{14mu}{OTF}_{shift}} = {\frac{F_{2}}{F_{1}} \cdot \frac{F_{1}}{F_{2}}}}{{{All}\text{-}{in}\text{-}{focus}\mspace{14mu}{image}\mspace{14mu}{of}\mspace{14mu}{subject}\text{:}\mspace{14mu}{F_{0}\left( \hat{d} \right)}} = \frac{{F_{1}\overset{\_}{{OTF}_{1}(d)}} + {F_{2}\overset{\_}{{OTF}_{2}(d)}{OTF}_{shift}}}{{{{OTF}_{1}(d)}}^{2} + {{{{OTF}_{2}(d)}{OTF}_{shift}}}^{2}}}} & \left( {{Formula}\mspace{14mu} 2} \right)\end{matrix}$

Distance determiner 132 uses the cost of the distance calculated by costcalculator 131 to determine the distance information to the subject foreach pixel. Here, among the plurality of distance d assumed for eachpixel, the distance d in which the cost of the distance in each pixelbecomes the smallest is output as the distance information of the pixel.That is, distance information calculator 130 calculates the distanceinformation to the subject for each pixel of the images.

FIG. 3 is a diagram illustrating difference in accuracy for positionaldeviation between distance information calculated by a conventionaltechnique and distance information calculated by a technique of thepresent disclosure.

The first image in which a subject having different distances in 16tones is captured and the second image in which the subject is deviatedvertically (Y in FIG. 3) and horizontally (X in FIG. 3) (an amount ofdeviation is shown by a numeral in FIG. 3, the unit is a number ofpixels) are used, and distance information calculated with theconventional technique and distance information calculated with thetechnique of the present disclosure are visually illustrated in FIG. 3.As illustrated in. FIG. 3, in the conventional technique, the distanceinformation does not represent the distance to the subject when theimage is deviated by only one pixel; however, in the technique of thepresent disclosure, the distance information accurately represents thedistance to the subject even when the pixel is deviated by one pixelvertically and horizontally.

As illustrated in this result, it shows that the distance information tothe subject cannot be accurately calculated unless alignment isaccurately performed up to a sub-pixel level, in the conventionaltechnique. On the other hand, in the technique of the presentdisclosure, it shows that the distance information to the subject can beaccurately calculated even in rough alignment.

In addition, in the conventional technique, when the subject is moved byabout one pixel during a period from acquisition of the first image toacquisition of the second image, the distance information to the subjectcannot be accurately calculated. On the other hand, in the technique ofthe present disclosure, even when the subject is moved by about onepixel during a period from acquisition of the first image to acquisitionof the second image, the distance information to the subject can beaccurately calculated without position correction.

In this way, only the amplitude component is used and the distanceinformation to the subject is calculated, whereby the distanceinformation can be calculated fast while a calculation cost issuppressed. Further, in an image processing method of the presentdisclosure, even when there is positional deviation between theplurality of images respectively having focusing positions differentfrom each other, the images are converted from the image space into thefrequency space, and only the amplitude component not including positioninformation is used and the distance information is calculated. For thatreason, when the positional deviation is within a predetermined range,the distance information can be accurately calculated. Therefore, evenin a case of capturing a still image or capturing a moving image of thesubject moving at a high speed, the present disclosure can be applied.

Second Exemplary Embodiment

Hereinafter, a second exemplary embodiment will be described withreference to FIG. 4.

FIG. 4 is a block diagram illustrating a functional configuration of animage processing device in the second exemplary embodiment.Incidentally, in the present exemplary embodiment, the same referencenumerals are used for the same configurations as those in the firstexemplary embodiment, and descriptions of the configurations areomitted.

In the present exemplary embodiment, to further improve accuracy ofdistance information calculation, positional deviation between differentimages is corrected and then distance information is calculated.

Image processing device 100 in the present exemplary embodiment furtherincludes area divider 140 and area searcher 150. Area divider 140 is aprocessor for dividing a first image into a plurality of first smallareas. A size of each of the first small areas divided is notparticularly limited, and for example, the first area may be dividedinto the first small areas of four pixels×four pixels.

Area searcher 150 is a processor that searches for a correspondingsecond small area for each of the plurality of first small areasdivided, for another image other than the first image. That is, areasearcher 150 corrects the positional deviation between images bysearching for an image of the corresponding second small area in thesecond image for each of images of the first small areas of the firstimage and performing alignment.

Specifically, an image corresponding to each of the first small areas ofthe first image divided by area divider 140 is used, and the secondimage is searched for a corresponding image, and an area of a similarimage is set to a second small area. For a search algorithm, although itis not particularly limited, in the present exemplary embodiment, blockmatching is used as the search algorithm. Incidentally, as the searchalgorithm, when it is possible to search for a corresponding areabetween images, another method may be used.

The block matching searches for the corresponding second small area inthe second image, for each of the first small areas of the first image.In searching, a difference is calculated between each of the first smallareas of the first image and each of the areas of the second image, andan area in which the difference is the smallest is set to the secondsmall area. The difference is calculated based on a sum of brightnessdifferences for the area, for example.

Frequency converter 110 in the present exemplary embodiment converts theimage of each of the first small areas and the image of thecorresponding second small area into frequency. Amplitude extractor 120extracts an amplitude component for each of small areas that are each ofthe first small areas, and the corresponding second small area. That is,amplitude extractor 120 extracts the amplitude component for each of thefirst small areas and for each corresponding second small area. Distanceinformation calculator 130 calculates distance information to a subjectfor each of the small areas. That is, distance information calculator130 calculates the distance information to the subject for each of thefirst small areas each aligned with the corresponding second small area.

As described above, in the present exemplary embodiment, the first imageof a plurality of images is divided into the first small areas, andalignment with another image is performed for each of the first smallareas, whereby the positional deviation between the images can be finelycorrected. For that reason, the distance information to the subject canbe calculated more accurately. However, in the present disclosure, sincethe distance information can be accurately calculated even when somepositional deviation occurs between the plurality of images, alignmentbetween the images can be performed roughly.

For example, in normal DFD, the distance information can be accuratelycalculated only when alignment is performed up to a unit of 0.1 pixelsfor an area with a weak edge. On the other hand, in the presentdisclosure, the distance information can be accurately calculated evenin alignment of about a unit of one pixel.

Third Exemplary Embodiment

Hereinafter, a third exemplary embodiment will be described withreference to FIG. 5.

FIG. 5 is a block diagram illustrating a functional configuration of adistance information calculator in the present exemplary embodiment.

As illustrated in FIG. 5, distance information calculator 130 includesrepresentative cost calculator 133 instead of cost calculator 131, andincludes representative distance determiner 134 instead of distancedeterminer 132.

Representative cost calculator 133 uses an amplitude component of eachof first small areas of a first image, an amplitude component of asecond small area of a second image, and input lens blur data tocalculate a representative cost of a distance assumed, for each of thesmall areas. That is, representative cost calculator 133 calculates therepresentative cost of the distance assumed, for each of the first smallareas each aligned with the corresponding second small area. Therepresentative cost of the distance is calculated with Formula 3 below.

$\begin{matrix}{\left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 3} \right\rbrack\mspace{365mu}} & \; \\{{{Cost}(d)} = {\sum\left\{ \frac{{{{{{OTF}_{2}(d)} \cdot {F_{1}}} - {{{OTF}_{1}(d)} \cdot {F_{2}}}}}^{2}}{{{{OTF}_{1}(d)}}^{2} + {{{OTF}_{2}(d)}}^{2}} \right\}}} & \left( {{Formula}\mspace{14mu} 3} \right)\end{matrix}$

Although a calculation technique for the representative cost of thedistance is not particularly limited, in the present exemplaryembodiment, one representative cost of the distance is calculated forone small area. Representative distance determiner 134 uses therepresentative cost of the distance calculated by representative costcalculator 133 to determine representative distance information for eachof the small areas. Specifically, representative cost calculator 133adds up frequency coefficient values, and representative distancedeterminer 134 calculates one piece of representative distanceinformation for one small area as a representative value.

As described above, in the present exemplary embodiment, therepresentative distance information is calculated for each of the smallareas, not for each pixel, whereby a calculation cost and memory usedcan be significantly reduced. For that reason, fast processing can beachieved, and a device cost can be reduced.

Specific effects of the present disclosure are, for example, as follows.

While a plurality of images respectively having focusing positionsdifferent from each other is captured, when positional deviation of asubject occurs between the plurality of images due to a camera shake ormovement of the subject, distance information to the subject cannot beaccurately obtained in some cases. In this case, alignment processing isrequired before calculating the distance information. However, in thealignment processing, due to a feature of the image, influence of noise,or the like, alignment cannot be accurately performed, or it takes along time to accurately perform alignment, in some cases.

In the present disclosure, when a plurality of images captured atdifferent timings is used and the distance information to the subject iscalculated, the distance information to the subject can be accuratelycalculated even when alignment between the images is not performed oralignment is roughly performed.

As described above, as examples of the technique disclosed in thepresent disclosure, the first to third exemplary embodiments have beendescribed. However, the technique in the present disclosure is notlimited to these exemplary embodiments, and can also be applied to anexemplary embodiment to which modification, replacement, addition,omission, and the like are performed. In addition, a modificationobtained by applying various modifications that can occur to thoseskilled in the art without departing from the gist of the presentdisclosure, that is, the meanings indicated by the words described inthe claims to the above exemplary embodiments, is included in thepresent disclosure.

For example, when distance information in only a part of pixels isrequired for an image, the distance information in only the part of thepixels may be calculated. Specifically, for example, it is not necessaryto perform searching to all first small areas, and may perform searchingto each of the first small areas of interest to calculate the distanceinformation.

In addition, the first small areas divided do not have to be completelyindependent of each other, and may partially lie on top of one another(overlap each other).

In addition, the shape of each of the first small areas is not limitedto a square, and any shape can be selected.

In addition, in the first to third exemplary embodiments, the OTF isused as an input; however, in general, the PSF is converted to the OTFby Fourier transform, so that the PSF may be used as the input, andFourier transform may be performed internally to convert the PSF to theOTF.

In addition, a program for causing a computer to execute imageprocessing methods included in image processing device 100 and arecording medium in which the program is recorded are also within thescope of the present disclosure.

INDUSTRIAL APPLICABILITY

The present disclosure can be applied to an image processing device andan image processing method that use a plurality of images respectivelyhaving focusing positions different from each other to calculatedistance information to a subject. Specifically, the present disclosurecan be applied to a digital still camera, a digital movie camera, amobile phone with a camera function, a smart phone, and the like.

REFERENCE MARKS IN THE DRAWINGS

-   -   100 image processing device    -   110 frequency converter    -   120 amplitude extractor    -   130 distance information calculator    -   131 cost calculator    -   132 distance determiner    -   133 representative cost calculator    -   134 representative distance determiner    -   140 area divider    -   150 area searcher

The invention claimed is:
 1. An image processing device that uses aplurality of images respectively having focusing positions differentfrom each other to calculate distance information to a subject, theimage processing device comprising: a frequency converter that convertsthe plurality of images into frequency; an amplitude extractor thatextracts an amplitude component out of a phase component and theamplitude component of a coefficient obtained by converting theplurality of images into frequency; and a distance informationcalculator that calculates the distance information, by using lens blurdata and only the amplitude component extracted by the amplitudeextractor out of the phase component and the amplitude component of thecoefficient, and outputs the calculated distance information forcontrolling a lens system of a camera in capturing of images., whereinthe distance information calculator includes: a cost calculator thatcalculates a cost of a distance assumed for the plurality of images,based on a depth calculation formula, by using the lens blur data andthe amplitude component extracted by the amplitude extractor, an imagedeviation being applied to one of the plurality of images and introducedinto the depth calculation formula; and a distance determiner thatdetermines the distance information, by using the cost of the distancecalculated by the cost calculator.
 2. The image processing deviceaccording to claim 1, further comprising: an area divider that dividesone image of the plurality of images into a plurality of first smallareas; and an area searcher that searches for a second small areacorresponding to each of the plurality of first small areas, for anotherimage of the plurality of images, wherein the frequency converterconverts an image of each of the plurality of first small areas, and animage of the second small area corresponding to each of the plurality offirst small areas into frequency, the amplitude extractor extracts theamplitude component, for each of the plurality of first small areas, andthe second small area corresponding to each of the plurality of firstsmall areas, and the distance information calculator calculates thedistance information, for each of the plurality of first small areas. 3.The image processing device according to claim 2, wherein the costcalculator calculates a representative cost of a distance assumed foreach of the plurality of first small areas, and the distance determinerdetermines representative distance information for each of the pluralityof first small areas, by using the representative cost of the distancecalculated by the cost calculator.
 4. The image processing deviceaccording to claim 1, wherein the distance information calculatorcalculates the distance information for each pixel of the plurality ofimages.
 5. An image processing method that uses a plurality of imagesrespectively having focusing positions different from each other tocalculate distance information to a subject, the image processing methodcomprising: converting the plurality of images into frequency;extracting an amplitude component out of a phase component and theamplitude component of a coefficient obtained by converting theplurality of images into frequency; and calculating the distanceinformation, by using lens blur data and only the amplitude componentextracted out of the phase component and the amplitude component of thecoefficient, wherein the calculating of the distance informationincludes: calculating a cost of a distance assumed for the plurality ofimages, based on a depth calculation formula, by using the lens blurdata and the amplitude component extracted, an image deviation beingapplied to one of the plurality of images and introduced into the depthcalculation formula; and determining the distance information, by usingthe cost of the distance calculated.
 6. An image processing device thatuses a plurality of images respectively having focusing positionsdifferent from each other to calculate distance information to asubject, the image processing device comprising: a frequency converterthat converts the plurality of images into frequency; an amplitudeextractor that extracts an amplitude component out of a phase componentand the amplitude component of a coefficient obtained by converting theplurality of images into frequency; a distance information calculatorthat calculates the distance information, by using lens blur data andonly the amplitude component extracted by the amplitude extractor out ofthe phase component and the amplitude component of the coefficient, andoutputs the calculated distance information for controlling a lenssystem of a camera in capturing of images; an area divider that dividesone image of the plurality of images into a plurality of first smallareas; and an area searcher that searches for a second small areacorresponding to each of the plurality of first small areas, for anotherimage of the plurality of images, wherein the frequency converterconverts an image of each of the plurality of first small areas, and animage of the second small area corresponding to each of the plurality offirst small areas into frequency, the amplitude extractor extracts theamplitude component, for each of the plurality of first small areas, andthe second small area corresponding to each of the plurality of firstsmall areas, and the distance information calculator calculates thedistance information, for each of the plurality of first small areas,wherein the distance information calculator includes: a cost calculatorthat calculates a cost of a distance assumed for the plurality ofimages, based on a depth calculation formula, by using the lens blurdata and the amplitude component extracted by the amplitude extractor,an image deviation being applied to one of the plurality of images andintroduced into the depth calculation formula; and a distance determinerthat determines the distance information, by using the cost of thedistance calculated by the cost calculator.